Tensor register files

ABSTRACT

Tensor register files in a hardware accelerator are disclosed. An apparatus may comprise tensor operation calculators each configured to perform a type of tensor operation. The apparatus may also comprises tensor register files, each of which is associated with one of the tensor operation calculators. The apparatus may also comprises logic configured to store respective ones of the tensors in the plurality of tensor register files in accordance with the type of tensor operation to be performed on the respective tensors. The apparatus may also control read access to tensor register files based on a type of tensor operation that a machine instruction is to perform.

BACKGROUND

Computing is increasingly requiring extremely powerful processors. Forexample, machine learning such as, but not limited to, deep neuralnetworks, requires a processor capable of performing an extremely highnumber of operations per second. Executing machine learning such as, butnot limited to, deep neural networks, on a general-purpose centralprocessing unit (CPU) can be extremely expensive.

Hardware accelerators have been used to supplement the processingperformed on general-purpose CPUs.

SUMMARY

Certain embodiments described herein relate to a hardware acceleratorhaving tensor register files. In one embodiment, an apparatus comprisestensor operation calculators each configured to perform a type of tensoroperation. The apparatus also comprises tensor register files, each ofwhich is associated with one of the tensor operation calculators. Theapparatus also comprises logic configured to store respective ones ofthe tensors in the plurality of tensor register files in accordance withthe type of tensor operation to be performed on the respective tensors.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts one embodiment of a hardware accelerator.

FIG. 2 depicts further details of one embodiment of a hardwareaccelerator.

FIG. 3A shows an example of a chain of machine instructions.

FIG. 3B is a flowchart of one embodiment of a process executing a chainof machine instructions in a tensor processor.

FIG. 3C is a diagram of further details of one embodiment of aninstruction decoder.

FIG. 4 is a flowchart of one embodiment of a process of executingmachine instructions in a hardware accelerator.

FIG. 5 is a flowchart of one embodiment of a process of executing amachine instruction.

FIGS. 6A-6D are flowcharts of embodiments of processes of differenttechniques that may be used to use a resultant tensor from one machineinstruction as a missing tensor operand in another machine instruction.

FIG. 7 is a flowchart of one embodiment of a process of accessing atensor from a tensor register file.

FIG. 8 is a flowchart of one embodiment of a process of storing tensorsin tensor register files.

FIG. 9 is a flowchart of one embodiment of a process of storing tensorsin a tensor processor in response to a machine instruction to store atensor in a tensor processor.

FIG. 10 is a flowchart of one embodiment of a process of accessingtensors from tensor register files in a tensor processor in which accessto the tensor register files is controlled based on a type of tensoroperation of a machine instruction.

FIG. 11 is a diagram that illustrates functionality that may be achievedby executing the code in Table II in one embodiment of a hardwareaccelerator.

FIG. 12 is a flowchart of one embodiment of a process of executing achain of machine instructions in a tensor processor based on a tilingfactor.

FIG. 13 illustrates an example environment in which embodiments of ahardware accelerator as described herein can operate.

DETAILED DESCRIPTION

Certain embodiments of the present technology relate to hardwareaccelerators. Some embodiments relate to tensor register files in ahardware accelerator. Some embodiments of the present technology relateto an instruction set architecture in a hardware accelerator.

A hardware accelerator may have various tensor operation calculators toperform various types of tensor operations. The tensor operationcalculators may need tensors to operate on. If a tensor operationcalculator has to wait for the needed tensor to be provided, computingefficiently may be significantly slowed.

In some embodiments, rather than having a large tensor register filethat stores tensors for all types of tensor operations, a hardwareaccelerator has multiple smaller tensor register files. These smallertensor register files may be dedicated to one instance of a tensoroperation calculator. Such a configuration may improve bandwidth bykeeping the various tensor operation calculators busy a high percentageof the time. For example, having the dedicated tensor register files mayavoid or at least reduce stalls that could otherwise occur while atensor operation calculator waits for a tensor to be provided fromstorage. Also, having multiple smaller tensor register files may reducecost versus having a larger centralized tensor register file that ismulti-ported to allow simultaneous access to different tensor operationcalculators.

A hardware accelerator may receive machine instructions in order toimplement, for example, a machine learning algorithm. For example, thehardware accelerator could receive machine instructions to cause thehardware accelerator to implement a deep neural network. It can take anextreme amount of memory to represent a machine learning algorithm, suchas a deep neural network. Decoding the machine instructions couldpotentially become a bottleneck, thereby reducing efficiency.

In some embodiments, the number of bits in a machine instruction used ina hardware accelerator is reduced to provide for efficient computing.This can provide a substantial memory savings. This can also save timein decoding the machine instructions. In some embodiments, advantage istaken of the observation that a tensor that results from execution ofone machine instruction is frequently used as an input tensor to thenext machine instruction.

One embodiment of a hardware accelerator has an instruction set in whichsome machine instructions are missing a tensor operand needed to executethe machine instruction. Therefore, the machine instruction takes fewerbits to encode. The hardware accelerator may execute a first machineinstruction, resulting in a tensor. The hardware accelerator may executea second machine instruction using the resultant tensor as a missingtensor operand in the second machine instruction. Thus, the hardwareaccelerator may take advantage of an observation that a tensor thatresults from execution of one machine instruction may frequently be usedas an input tensor to another machine instruction. Also, using theresults of the first machine instruction for a tensor operand in thesecond machine instruction obviates the need to store the results of thefirst machine instruction in a register file in the hardware accelerator(or other storage such as memory external to the hardware accelerator),as well as reading the results back out from the storage in a separatemachine instruction. Thus, hardware resources may be saved and latencybetween machine instructions may be reduced.

FIG. 1 depicts one embodiment of a hardware accelerator 100. Thehardware accelerator 100 may also be referred to herein as a tensorprocessor. The hardware accelerator 100 may be implemented in a fieldprogrammable gate array (FPGA), application specific circuit (ASIC),application specific standard product (ASSP), system on a chip (SoC),complex programmable logic device (CPLD), but is not limited thereto.The term “hardware” accelerator broadly encompasses different ways ofleveraging a hardware device to perform a function, including, forinstance, at least: a) a case in which at least some tasks areimplemented in hard ASIC logic or the like; b) a case in which at leastsome tasks are implemented in soft (configurable) FPGA logic or thelike; c) a case in which at least some tasks run as software on FPGAsoftware processor overlays or the like; d) a case in which at leastsome tasks run as software on hard ASIC processors or the like, and soon, or any combination thereof.

The hardware accelerator 100 includes several tensor operationcomponents 102(1)-102(3), a tensor memory manager 104, an instructionqueue 120, and an instruction decoder 106. The instruction queue 120 mayreceive machine instructions from outside the hardware accelerator 100to be performed by the hardware accelerator 100. In one embodiment, thehardware accelerator 100 resides in an environment such as environment1300 in FIG. 13. In that environment, the hardware accelerator 100 couldreceive machine instructions over bus 1314. For example, processor(s)1306 could send machine instructions to the hardware accelerator 100.

The instruction decoder 106 may be configured to decode the machineinstructions and to send control signals to other elements in thehardware accelerator 100 in order to execute the machine instructions.In some embodiments, the control signals are referred to assub-component commands.

The tensor memory manager 104 may receive tensors to be used by thehardware accelerator 100 when executing the machine instructions. Thetensors may originate from memory that is external to the hardwareaccelerator 100. For example, with reference to FIG. 13, the tensors mayoriginate from memory 1312. The tensor memory manager 104 may also storetensors to memory external to the hardware accelerator 100, such asmemory 1312.

Herein, a tensor includes, but is not limited to, vectors and matrices.A vector, as defined herein, is comprised of an ordered collection of“v” elements, where “v” is an integer greater than one. A matrix, asdefined herein, is comprised of an array of “n×m” elements, where “n”and “m” are each integers greater than one. The matrix could be a squarematrix (in which case n=m), but is not required to be a square matrix. Avector may correspond to either a row or a column of a matrix, in somecases. For example, the value of “v” for a given vector could be equalto “n” and/or to “m”. However, the value of “v” is not required to beequal to either “n” or “m” (in which case the vector may be used forother than a matrix-vector operation). Note that it is not required thatthe value of “v” be the same for all vectors stored and operated on bythe hardware accelerator 100. Likewise, it is not required that thevalues of “n” and “m” be the same for all matrices stored and operatedon by the hardware accelerator 100. The tensor elements are scalarvalues, in some embodiments. It will be understood that the term“tensor”, as used herein, does not require the tensors to obey anytransformation rules.

The tensor operation components 102(1)-102(3) each may be configured toperform one or more types of tensor operations. Thus, each tensoroperation component 102 may be configured to perform a set of one ormore types of tensor operations. A tensor operation may involve one ormore tensors. For example, some tensor operations involve a singletensor, some involve two tensors. In some embodiments, a tensoroperation involves three or more tensors. Examples of the types oftensor operations involving two tensors include, but are not limited to,matrix-matrix multiply, matrix-vector multiply, matrix inversion,vector-vector multiply, vector-vector addition, and vector-vectorsubtraction. Note that for some of these tensor operations, there may bemore than one technique. For example, vector-vector multiply could be adot product or a cross product.

To perform the various types of tensor operations, the tensor operationcomponents 102 have various tensor operation calculators 112. Tensoroperation component 102(1) has tensor operation calculator 112(1).Tensor operation components 102(2) and 102(3) each have an instance oftensor operation calculator 112(2), as well as an instance of tensoroperation calculator 112(3). Tensor operation calculator 112(1) performssome type of tensor operation, such as matrix-vector multiply. Tensoroperation calculator 112(2) performs another type of tensor operation,such as vector-vector multiply. Tensor operation calculator 112(3)performs still another type of tensor operation, such as vector-vectoraddition. Note that some tensor operation calculators 112 could performmore than one type of tensor operation. For example, tensor operationcalculator 112(3) could also perform as vector-vector subtraction. Theremay be other tensor operation calculators 112 that perform other typesof tensor operations.

Three tensor operation components 102 are depicted, but there may bemore or fewer than three tensor operation components 102. In some cases,two or more of the tensor operation components 102 may be configured toperform the same set of tensor operations. In the example of FIG. 1,tensor operation component 102(2) and 102(3) are each able to performthe same set of tensor operations. However, it is not required to havemore than one tensor operation component 102 that is able to perform thesame set of tensor operations.

In some embodiments, rather than having a large tensor register filethat stores tensors for all types of tensor operations, the hardwareaccelerator 100 has multiple smaller tensor register files 110, at leastsome of which are used to store tensors for a specific type of tensoroperation. These smaller tensor register files 110 may be dedicated toone instance of a tensor operation calculator 112. Such a configurationmay reduce cost versus having a larger centralized tensor register filethat is multi-ported to allow simultaneous access to different tensoroperation calculators 112. Also, this configuration may improvebandwidth by keeping the various tensor operation calculators 112 busy ahigher percentage of the time. For example, having the dedicated tensorregister files 110 may avoid or at least reduce stalls that couldotherwise occur while a tensor operation calculator 112 waits for atensor to be provided from storage.

The hardware accelerator 100 includes a number of tensor register files110(1)-110(5). The tensor register files 110 are used to store tensorsfor inputs to the tensor operation calculators 112. In one embodiment,at least some of the tensor register files (e.g., 110(1)-110(4) arededicated to a particular type of tensor operation calculator 112. Notethat it is possible for one or more tensor register files (e.g., 110(5))to not be dedicated to a particular type of tensor operation calculator112. Thus, at least some of the tensor register files 110 may bededicated to a certain set of one or more tensor operations. Forexample, tensor register file 110(1) may be used to store tensors to beinput to tensor operation calculator 112(1). Likewise, tensor registerfile 110(4) may be used to store tensors to be input to tensor operationcalculator 112(1). In one embodiment, whenever the tensor memory manager104 receives a tensor that is to be used in a type of tensor operationthat is performed by tensor operation calculator 112(1), that tensor maybe stored in tensor register file 110(1) or 110(4).

In the example of FIG. 1, tensor register file 110(2) may be used tostore tensors to be input to tensor operation calculator 112(2). Notethat there are two instances of tensor register file 110(2) and twoinstances of tensor operation calculator 112(2). Each instance of tensoroperation calculator 112(2) has its own dedicated tensor register file110(2), in one embodiment. In one embodiment, whenever the tensor memorymanager 104 receives a tensor that is to be used in the type of tensoroperation that is performed by tensor operation calculator 112(2), thattensor is stored in all instances of tensor register file 110(2). Notethat a tensor operation calculator 112 may have zero, one, two, or morededicated tensor register files 110.

In the example of FIG. 1, tensor register file 110(3) may be used tostore tensors to be input to tensor operation calculator 112(3). Notethat there are two instances of tensor register file 110(3) and twoinstances of tensor operation calculator 112(3). Each instance of tensoroperation calculator 112(3) has its own dedicated tensor register file110(3), in one embodiment. In one embodiment, whenever the tensor memorymanager 104 receives a tensor that is to be used in the type of tensoroperation that is performed by tensor operation calculator 112(3), thattensor is stored in all instances of tensor register file 110(3).

Tensor register file 110(5) is not necessarily dedicated to one of thetensor operation calculators 112. Tensor register file 110(5) isconfigured to provide tensors to tensor operation component 102(2).Thus, the tensors in the tensor register file 110(5) could be used byany of the tensor operation calculators 112(1)-112(3).

In one embodiment, the tensor register files 110(1)-110(4) representfour different types of tensor register files, each of which isassociated with a different set of one or more types of tensoroperations. For example, tensor register file 110(1) may be associatedwith a first set of one or more types of tensor operations, eachinstance of tensor register file 110(2) may be associated with a secondset of one or more types of tensor operations, each instance of tensorregister file 110(3) may be associated with a third set of one or moretypes of tensor operations, and each instance of tensor register file110(4) may be associated with a fourth set of one or more types oftensor operations. The first through fourth sets of tensor operationsmay each include one or more types of tensor operations, with each setbeing different from the others.

Note that more than one of the tensor register files 110 can be accessedsimultaneously (for both read access and write access). This can helpallow the various tensor operation calculators 112 to remain busy, whichmakes efficient use of the resources in the hardware accelerator 100.For example, tensor operation calculator 112(1) may perform a readaccess of tensor register file 110(4) at the same time that the instancetensor operation calculator 112(2) in tensor operation component 102(2)performs a read access of tensor register file 110(2), and at the sametime that the instance tensor operation calculator 112(3) in tensoroperation component 102(2) performs a read access of tensor registerfile 110(3).

In one embodiment, the hardware accelerator 100 is configured to processchains of machine instructions. In one embodiment, the chain is anatomic unit of computation. For example, the chain may begin with amachine instruction to load a tensor from memory external to thehardware accelerator 100, and may end with a machine instruction tostore a tensor to memory external to the hardware accelerator 100.However, no other machine instructions in the chain loads or storetensors to or from memory, in one embodiment. Further details of oneexample of a chain of machine instructions are shown and discussed withreference to FIG. 3A.

FIG. 2 depicts further details of one embodiment of a hardwareaccelerator 100. The hardware accelerator 100 may also be referred to asa tensor processor. FIG. 2 shows one possible configuration; however,note that the hardware accelerator 100 could have many differentconfiguration. The configuration may be tailored to the types ofoperations that the hardware accelerator 100 is expected to perform.This allows the hardware accelerator 100 to be extremely efficient. Theconfiguration of FIG. 2 may be used to perform operations (orcalculations) in a deep neural network, but is not limited to deepneural networks.

In the embodiment of FIG. 2, tensor operation component 102(1) mayperform a matrix-vector multiply. Thus, tensor register file 110(1) maybe used to store matrices to be used in the matrix-vector multiply.Tensor register file 110(4) may be used to store vectors to be used inthe matrix-vector multiply.

In the embodiment of FIG. 2, tensor operation component 102(1) includesa format converter 214 and a format de-converter 212. The formatconverter 214 may be used to convert a format of the input tensor to onethat is more suitable for the matrix-vector multiply. In one embodiment,the format converter 214 performs a conversion from floating point toblock floating point. In one embodiment, the format de-converter 212performs a conversion from block floating point back to floating point.Note that the tensors throughout the hardware accelerator 100 are notlimited to a particular format. For example, scaler values in theelements of a tensor could be represented in fixed point, floatingpoint, block floating point, etc.

In the embodiment of FIG. 2, tensor operation component 102(2) has acrossbar component 236. The crossbar component 236 is configured toroute tensors that are input to the tensor operation component 102(2) toa suitable tensor operation calculator 112(2), 112(3), 112(4). Thecrossbar component 236 may also be configured to route tensors fromtensor operation calculator 112 in tensor operation component 102(2) toanother tensor operation component 102(2). The crossbar component 236could include multiplexers and de-multiplexers, which respond to controlsignal from the instruction decoder 106 to route the tensors.

Note that a tensor operation calculator 112 may have a dedicated tensorregister file 110. For example, tensor register file 110(2) may bededicated to tensor operation calculator 112(2), and tensor registerfile 110(3) may be dedicated to tensor operation calculator 112(3).

Tensor operation calculator 112(4) is configured to perform an operationon a single tensor. One example is to take the sigmoid of each elementin the tensor. Another example is to take the hyperbolic tangent of eachelement in the tensor. Another example is to perform a “ReLU” on eachelement in the tensor. A ReLU refers to a rectified linear unit. Anotherexample is to raise each element in the tensor to the power of ex. Notethat tensor operation calculator 112(4) could be broken into differentcalculation components, or a single calculation component may be able toperform multiple types of calculations.

Also note that some tensor calculation components 112 input two tensoroperands, whereas other tensor calculation components input a singletensor operand. For example, tensor operation calculator 112(2) has twoinputs for tensor operands (as indicated by the two input arrows).Likewise, tensor operation calculator 112(3) has two inputs for tensoroperands (as indicated by the two input arrows). However, tensoroperation calculator 112(4) has only input for tensor operands (asindicated by the single input arrow. Each tensor calculation componentin tensor operation component 102(2) outputs a tensor to the crossbarcomponent 236. The crossbar component 236 in tensor operation component102(2) outputs tensors to tensor operation component 102(3).

Tensor operation component 102(3) has similar components and operationas just discussed with respect to tensor operation component 102(2). Adifference being that tensor operation component 102(3) is “downstream”from tensor operation component 102(2).

The instruction decoder 106 receives machine instructions from theinstruction queue 120, decodes the machine instructions, and issuescontrol signals to other components in the hardware accelerator 200. Thecontrol signals may also be referred to as sub-component commands. Thecontrol signals are represented in FIG. 2 by dashed arrows emanatingfrom the instruction decoder 106.

The instruction decoder 106 sends control signals (or sub-componentcommands) to each of the tensor operation components 102(1)-102(3). Thecontrol signals include sub-component commands that instruct the tensoroperation components 102 how to operate on the tensors in the tensorregister files 110. Together, the tensor operation components mayoperate as a pipeline to implement a chain of instructions.

The hardware accelerator 100 includes several multiplexers 226, 228, 230and two de-multiplexers 224 and 232, which are used to control the flowof tensors. The instruction decoder 106 sends control signals to themultiplexers 226, 228, 230 and de-multiplexers 224 and 232 to route thetensors. The instruction decoder 106 also sends a control signal tomulti-function initial tensor component 234, which provides tensors tomultiplexer 230.

Multiplexer 230 receives tensors output from tensor operation component102(1), and a tensor from tensor register file 110(5) in themulti-function initial tensor component 234. For example, multiplexer230 has a first input that receives a first tensor from tensor operationcomponent 102(1) and a second input that receives a second tensor fromtensor register file 110(5) in the multi-function initial tensorcomponent 234. Multiplexer 230 outputs one of the tensors to tensoroperation component 102(2) responsive to a control signal frominstruction decoder 106. In this example, the tensor is provide tocrossbar 236 in tensor operation component 102(2).

De-multiplexer 232 receives a tensor output from tensor operationcomponent 102(3). That tensor is routed to one or more of output queue222, multiplexer 226, and/or multiplexer 228 responsive to a controlsignal from instruction decoder 106.

The tensor memory manager 104 includes an input queue 202, matrix router204, and vector router 206. The input queue gates the flow of tensorsinto the hardware accelerator 200. The input queue 202 may be used tostore tensors as they are first provided to the hardware accelerator 200from memory external to the hardware accelerator 200 such as memory 1312(see FIG. 13). These tensors may include matrices and/or vectors.

The tensor memory manager 104 manages the flow of tensors from the inputqueue 202 to various elements in the hardware accelerator 200. Thetensor memory manager 104 may receive control signals from theinstruction decoder 106, to instruct the tensor memory manager 104 as tothe routing of the tensors. The matrix router 204 is configured to routematrices from the input queue 202 to tensor register file 110(1). Thus,in this embodiment, each entry in tensor register file 110(1) may storea matrix. The vector router 206 is configured to route vectors from theinput queue 202 to de-multiplexer 224. De-multiplexer 224 providesvectors to multiplexer 226 and to multiplexer 228, in response tocontrol signals from the instruction decoder 106.

Thus, multiplexer 226 has a first input that receives a first vectorfrom de-multiplexer 224 and a second input that receives a second vectorfrom de-multiplexer 232. Multiplexer 226 has an output that is used toprovide a vector to tensor register file 110(4) and/or 110(5) responsiveto a control signal from the instruction decoder 106. Tensor registerfile 110(4) is in tensor operation component 102(1). Tensor registerfile 110(5) is in multi-function initial tensor component 234. In oneembodiment, both tensor register files 110(4) and 110(5) are used tostored “initial” tensors at/near the beginning of chain of instructions.

Multiplexer 228 has a first input that receives a first vector fromde-multiplexer 224 and a second input that receives a second vector fromde-multiplexer 232. Multiplexer 228 has an output that is used toprovide a vector to tensor register files 110(2) and 110(3), responsiveto control signals from instruction decoder 106. In one embodiment,responsive to a control signal from instruction decoder 106, multiplexer228 provides one of its input vectors to all instances of tensorregister files 110(2). In one embodiment, responsive to a control signalfrom instruction decoder 106, multiplexer 228 provides one of its inputvectors to all instances of tensor register files 110(3).

Thus, multiplexer 228 may be used to route a vector to all instances ofa tensor register file that are associated with a particular type oftensor operation. For example, if a vector is to be used for avector-vector multiply, the instruction decoder 106 may send theappropriate control signals to route that vector to all instances oftensor register files that are dedicated to a vector-vector multiplier.As another example, if a vector is to be used for a vector-vectoraddition/subtraction, the instruction decoder 106 may send theappropriate control signals to route that vector to all instances oftensor register files that are dedicated to a vector-vectoradder/subtractor.

In one embodiment, tensor operation components 102(2) and 102(3) areconfigured to operate on vectors, but not on matrices. Note that thetensor operation components 102(2) and 102(3) could also be configuredto perform operate on matrices. In this case, multiplexers 226 and 228could provide matrices to the tensor register files 110 in the tensoroperation components 102(2) and 102(3). Thus, tensor memory manager 104could route matrices to de-multiplexer 224 instead of, or in addition torouting vectors to de-multiplexer 224.

The output queue gates the flow of tensors out the hardware accelerator200. For example, tensors may be stored to memory external to thehardware accelerator 200, such as memory 1312 (see FIG. 13).

In one embodiment, a tensor that results from one machine instructionexecuted in a hardware accelerator 100 (including but not limited to theembodiments of hardware accelerators 100 in FIGS. 1 and/or 2) is used asan input tensor for the next machine instruction in the chain. Toillustrate, FIG. 3A shows an example of a chain 300 of machineinstructions 302(1)-302(5). The example chain 300 starts with a“vector_read” machine instruction 302(1) to read a vector from location“x” in memory. In this example, the destination in which to load thevector is not expressly specified in the vector_read machine instruction302(1). In one embodiment, the vector that is loaded by the vector_readmachine instruction is used as an input to the next machine instructionin the chain. The result of executing the “vector_read” machineinstruction produces what is referred to herein as a “resultant tensor.”

The second machine instruction 302(2) is a “matrix_vector multiply”instruction to perform a matrix-vector multiply. This is one example ofa two tensor operation. Hence, two tensor operands are necessary forthis machine instruction. Note that only one tensor operand (“Wx”) isspecified. The “XXX” indicates that one of the tensor operands is notspecified in the machine instruction 302(2). Herein, this is referred toas a “missing tensor operand.” In this example, the resultant tensorfrom the “vector_read” machine instruction 302(1) may be used for themissing tensor operand.

The third machine instruction 302(3) is a “vector_vector add”instruction to add two vectors. This is another example of a two tensoroperation. Hence, two tensor operands are necessary for this machineinstruction 302(3). Again, note that only one tensor operand (“b”) isspecified. Again, the “XXX” indicates that one of the tensor operands isnot specified in the machine instruction 302(3). In this example, theresultant tensor from the “matrix_vector multiply” machine instruction302(2) may be used for the missing tensor operand.

The fourth machine instruction 302(4) is a “vector_ReLU” instruction toperform a ReLU operation on a vector. This is an example of a singletensor operation. Hence, one tensor operand is necessary for thismachine instruction 302(4). Note that no tensor operands are specified.Again, the “XXX” indicates a missing tensor operand. In this example,the resultant tensor from the “vector_vector add” machine instruction302(3) may be used for the missing tensor operand.

The fifth machine instruction 302(5) is a “vector_write” instruction towrite a vector to location “h” in memory. However, the vector to bewritten to memory is not specified. The “XXX” indicates a missing tensoroperand. In this example, the resultant tensor from the “vector_ReLU”machine instruction 302(4) may be used for the missing tensor operand.

In one embodiment, the chain 300 is used to model a neuron in a deepneural network. In one embodiment, the chain 300 contains a machineinstruction that serves as an activation function. The term “activationfunction” is well-understood by those in the field of deep neuralnetworks. In FIG. 3A, machine instruction 302(4) serves as an activationfunction. Other possible activation functions include, but are notlimited to, sigmoid and hyperbolic tangent.

One observation to be made of the chain of instructions is that theinitial tensor to be used may be read in from memory by the vector_readcommand. As another example, a “matrix_read” instruction may be used toread in an initial matrix to a tensor register file. Another observationto be made is that the chain ends with an instruction to store a vectorto memory. However, there are no memory accesses between the initialvector read and the vector write. Hence, the chain may be considered tobe an atomic unit of computation.

In one embodiment, the beginning of the chain is defined by a read frommemory, such as machine instruction 302(1), and the end of the chain isdefined by a write to memory, such as machine instruction 302(5).However, there could be another instruction prior to the read frommemory, such as the tiling factor to be discussed below (see FIG. 12,for example). Also, one option is for a machine instruction to expresslydefine the start or end of the chain, such as a “chain_begin”instruction or a “chain_end” instruction. For example, if the chain doesnot store a tensor to memory at the end, then a “chain_end” instructionmight be used to indicate the end of the chain.

Another observation to be made of the chain of instructions is that forthe first machine instruction, the tensor that results from executingthe machine instruction is used as a tensor input to the followingmachine instruction. The middle machine instructions each receive atensor input from the immediate prior machine instruction and provide atensor to the immediate following machine instruction. The final machineinstruction receives a tensor from the immediate prior machineinstruction. This pattern is one example, and there may be somedeviation from this pattern in other chains.

Another observation to be made of the chain of instructions is thatthose machine instructions that receive a tensor operand from anothermachine instructions can be coded with fewer bits than if the tensoroperand were expressly specified. For example, the matrix_vectormultiply machine instruction can be coded with a one byte opcode(“matrix_vector multiply”) and a one byte operand (“Wx”). Not specifyingthe missing operand may save one byte. Similar reasoning applies toother machine instruction that have a missing tensor operand. This savesa considerable amount of memory. Note that expressing a program such asa deep neural network into machine code may require an extreme amount ofmemory. Hence, reducing the number of bits in a machine instruction canprovide a substantial memory savings. There may also be time saved indecoding the machine instructions by the instruction decoder 106.

In one embodiment, the instruction decoder 106 is configured to sendcontrol signals to cause the chain of machine instructions to execute.For example, the instruction decoder 106 may cause the initial tensor tobe loaded into tensor register file 110(4) (“vector_read”). Theinstruction decoder 106 may cause tensor operation calculator 112(1) toperform a matrix-vector multiply using a matrix from tensor registerfile 110(1) and a tensor from tensor register file 110(4) in tensoroperation component 102(1) (“matrix_vector multiply”). The instructiondecoder 106 may cause tensor operation calculator 112(2) in tensoroperation component 102(2) to perform a vector-vector multiply using thetensor that results from the matrix-vector multiply in tensor operationcalculator 112(1) and a tensor in tensor register file 110(2) in tensoroperation component 102(2) (“vector_vector add”). The instructiondecoder 106 may cause tensor operation calculator 112(4) in tensoroperation component 102(2) to perform a vector ReLU using the tensorthat results from the vector-vector multiply (“vector_ReLU). Theinstruction decoder 106 may cause the tensor that results from thevector ReLU to be stored to memory external to the hardware accelerator100 (“vector_write”). This chain of machine instructions is just oneexample of a myriad of possible sequences.

In one embodiment, the hardware acceleration has a configuration that istailored to the way machine instructions are typically chained together.For example, referring to the example chain 300, the matrix_vectormultiply instruction 302(2) might be executed in tensor operationcalculator 112(1), with the resultant tensor being passed downstream totensor operation component 102(2). Tensor operation component 102(2) mayperform machine instructions 302(3) and 302(4). Some chains may haveadditional machine instructions that might be performed furtherdownstream in tensor operation component 102(3). The configuration maykeep the various tensor operation calculators 112 busy a very highpercentage of the time. In particular, tensor operation calculator112(1) might be kept occupied doing matrix_vector multiply a highpercentage of the time, which makes very efficient use of the hardwareresources.

FIG. 3B is a flowchart of one embodiment of a process 350 executing achain of machine instructions in a tensor processor 100. The process 350may be implemented in the tensor processor 100 of FIG. 1 or 2, but isnot limited thereto. Reference will be made to the example chain of FIG.3A, but this is just for purposes of discussion. In one embodiment, thechain implements a neuron in a deep neural network.

Step 352 includes a determination of whether a new chain has started. Inone embodiment, a machine instruction to read a tensor into the tensorprocessor 100 indicates that a new chain has started. In one embodiment,a machine instruction to alter a native size of tensors indicates that anew chain has started. FIG. 12 below provides further details of oneembodiment of a tiling factor that may be used to scale the native sizeof tensors. Referring to FIG. 3A, as one example, the “vector_read”machine instruction 302(1) indicates that a new chain has started, inthat example. Another option is for there to be a “begin_chain” machineinstruction. Another option is for there to be a “matrix_read” machineinstruction.

Step 354 includes executing the first machine instruction in the chainto produce a resultant tensor. Referring to FIG. 3A, as one example,executing the “vector_read” machine instruction 302(1) produces aresultant tensor by the act of storing the tensor into a tensor registerfile 110.

Step 356 includes executing the next machine instruction 302 in thechain 300 using the resultant tensor from the immediate prior machineinstruction. In one embodiment, the resultant tensor is used for amissing tensor operand. Examples of step 356 have been discussed abovein connection with FIG. 3A.

Step 358 is a determination of whether the chain has ended. With respectto the example chain of FIG. 3A, the chain ends after execution ofmachine instruction 302(5). Another possibility is for there to be amachine instruction of “chain_end” or the like.

The process 350 returns to step 356 to execute the next machineinstruction 302 in the chain 300 in the event that the chain has notended. As noted, the next machine instruction may use the resultanttensor from the immediate prior machine instruction. Thus, the variousmachine instructions 302 may be considered to be a “chain” in which thelink is formed by using the resultant tensor from one machineinstruction to execute the next machine instruction.

FIG. 3C is a diagram of further details of one embodiment of aninstruction decoder 106. The instruction decoder 106 may be used in thehardware accelerator 100 of FIG. 1 or FIG. 2, but is not limitedthereto. The instruction decoder 106 includes an operator decoder 312,an operand decoder 304, missing tensor operand determination logic 306,tensor register file determination logic 308, and sub-component commandprovider 310.

The operator decoder 312 is configured to decode the opcode portion ofmachine instructions 302. For example, with respect to the examplemachine instruction 302(1) of FIG. 3A, the “vector_read” portion may bespecified by a unique value of a one byte opcode. Each machineinstruction 302 may have a unique value for the one byte opcode. Theopcode could be smaller or large than one byte.

The operand decoder 304 is configured to decode the operand portion of amachine instruction (assuming there is an operand). For example, withrespect to the example machine instruction 302(1) of FIG. 3A, the “x”portion may be specified by a unique value of a one byte operand field.Some machine instructions do not have an operand field. For example, theembodiment of the vector_ReLU instruction 302(4) in FIG. 3A does notneed a field for an operand.

Some machine instructions, which require two tensor operands to executehave just a single operand field. For example, the embodiment of thematrix_vector multiple instruction 302(2) in FIG. 3A requires two tensoroperands, but only has a single tensor operand field (e.g., the “Wx”).Note that the “XXX” is used to represent that there are not any bits inmachine instruction 302(2) to specify one of the tensor operands neededto execute instruction 302(2).

The missing tensor operand determination logic 306 is configured todetermine a missing tensor operand for a machine instruction 302. Asnoted, some machine instructions may be missing a tensor operand that isneeded to execute the machine instruction. Some machine instructionsneed one tensor operand, but do not specify any tensor operands. Somemachine instructions need two tensor operands, but only specify a singletensor operand. In each case, the missing tensor operand determinationlogic 306 determines the missing tensor operand. In one embodiment, themissing tensor operand determination logic 306 assumes that the missingtensor operand is a tensor that results from a prior machine instructionin the chain.

The tensor register file determination logic 308 may determine whichtensor register file 110 should be accessed to store tensors. The tensorregister file determination logic 308 may also determine which tensorregister file 110 should be accessed when executing a particular type ofmachine instruction. In one embodiment, the tensor register file 110 isimplied by the type of machine instruction. In one embodiment, thetensor register file 110 for a read access is implied by the type oftensor operation that the machine instruction is to perform. Furtherdetails are discussed below.

The sub-component command provider 310 is configured to provide commands(or control signals) to the various sub-components in the hardwareaccelerator 100. For example, sub-component command provider 310provides commands to the tensor memory manager 104, the tensor operationcomponents 102, the various multiplexers 226, 230, 230, the variousde-multiplexers 224, 232, and the multi-function initial tensorcomponent 234.

FIG. 4 is a flowchart of one embodiment of a process 400 of executingmachine instructions in a hardware accelerator 100. The process 400 maybe executed within hardware accelerator 100 of FIG. 1 or 2, but is notlimited thereto. The process 400 involves using a tensor that resultsfrom one machine instruction in a set of machine instructions as aninput to another machine instruction in the set. In one embodiment, theset of machine instructions is a chain of machine instructions that arelinked by using a resultant tensor from executing one machineinstruction to execute the next instruction in the chain that needs atensor. In one embodiment, the chain implements a neuron in a deepneural network.

In step 402, a first machine instruction is accessed from a set ofmachine instructions. In step 404, a second machine instruction isaccessed from the set of machine instructions. The second machineinstruction involves a tensor that is not specified by the secondmachine instruction, in one embodiment. In one embodiment, the secondmachine instruction is missing a tensor operand.

In step 406, the first machine instruction is executed in the hardwareaccelerator 100 resulting in a tensor. In one embodiment, step 406includes loading a tensor into a tensor register file. For example, withrespect to FIG. 1 or 2, the tensor may be loaded into any of the tensorregister files 110(1)-110(5). Each of these loads results in a tensorbeing added to a tensor register file, which is referred to herein as a“resultant tensor.”

In one embodiment, step 406 includes performing a tensor operation in atensor operation calculator 112. This could be a single tensor operationor a two tensor operation. As one example of a single tensor operation,a tensor is input to tensor operation calculator 112(4) in tensoroperation component 102(2), which takes a sigmoid of each element in thetensor. As another example of a single tensor operation, a tensor isinput to tensor operation calculator 112(4) in tensor operationcomponent 102(2), which takes a hyperbolic tangent of each element inthe tensor. As another example of a single tensor operation, a tensor isinput to tensor operation calculator 112(4) in tensor operationcomponent 102(2), which takes a ReLU of each element in the tensor. Inall cases the result of the single tensor operation is what is referredto herein as a “resultant tensor”. In one embodiment, the resultanttensor is a vector. In one embodiment, the resultant tensor is a matrix.

As example of a two tensor operation, two tensors are input to tensoroperation calculator 112(1) in tensor operation component 102(1), whichperforms a matrix-vector multiplication. As another example of a twotensor operation, two tensors are input to tensor operation calculator112(2) in tensor operation component 102(2), which performs avector-vector multiplication. As another example of a two tensoroperation, two tensors are input to tensor operation calculator 112(3)in tensor operation component 102(2), which performs a vector-vectoraddition. As another example of a two tensor operation, two tensors areinput to tensor operation calculator 112(3) in tensor operationcomponent 102(2), which performs a vector-vector subtraction. In allthese two tensor cases the result of the two tensor operation is what isreferred to herein as a “resultant tensor”. In one embodiment, theresultant tensor is a vector. In one embodiment, the resultant tensor isa matrix.

In step 408, the second machine instruction is executed using theresultant tensor from the first machine instruction as the missingtensor operand.

Note that the resultant tensor of the second machine instruction may beused as an input to another machine instruction in the set. For example,there may be a third machine instruction in the set, which has a missingtensor operand. Thus, in one embodiment, a resultant tensor of executingthe second machine instruction is used to execute a third machineinstruction immediately following the second machine instruction inresponse to the third machine instruction missing a tensor operandneeded to execute the third machine instruction. This may continue onfor additional machine instructions in the set that are missing a tensoroperand. In one embodiment, the hardware accelerator 100 assumes thatwhenever there is a missing tensor operand in a machine instruction,that the resultant tensor of the immediate prior machine instruction inthe set be used for the missing tensor operand.

FIG. 5 is a flowchart of one embodiment of a process 500 of executing amachine instruction 302 in a chain of machine instructions. In oneembodiment, the chain implements a neuron in a deep neural network. Step502 includes accessing a machine instruction from a chain of machineinstructions. For example, instruction decoder 106 accesses aninstruction from instruction queue 120. For purposes of discussion, thiswill be referred to as the “subject machine instruction.”

Step 504 is a determination of whether the subject machine instructionfails to specify a tensor operand that is necessary to execute thetensor operation of the subject machine instruction. In one embodiment,instruction decoder 106 makes this determination. In the event thatthere is not a missing tensor operand, then circuitry in the hardwareaccelerator 100 executes the subject machine instruction using a tensorspecified in the machine instruction, in step 506.

If the subject machine instruction is a missing tensor operand, then theprocess 500 continues at step 508. Step 508 includes a determination ofwhether one or two tensor operands are needed to execute the subjectmachine instruction. In one embodiment, instruction decoder 106 makesthis determination. In process 500, it is assumed that at most onetensor operand is missing. In the event that one tensor operand isneeded for the subject machine instruction, then steps 510-512 areperformed.

In step 510, circuitry in the hardware accelerator 100 identifiesanother machine instruction that is to provide the missing tensoroperand. In one embodiment, instruction decoder 106 makes thisdetermination. In one embodiment, circuitry in the hardware acceleratorassumes that the missing tensor operand is to come from the machineinstruction that immediately precedes the subject machine instruction.An exception may be made if the immediately preceding instruction doesnot result in a tensor, in which case the hardware accelerator can gobackwards in the chain to look for a machine instruction capable ofproviding the missing tensor operand.

In step 512, a resultant tensor from the identified machine instructionis used as the missing tensor operand for the subject machineinstruction. In one embodiment, the crossbar component 236 supplies themissing tensor to tensor operation calculator 112(4). Note that themissing tensor operand might come from another tensor operationcalculator 112, such as tensor operation calculator 112(2) or tensoroperation calculator 112(3). The crossbar component 236 may act inresponse to a control signal from the instruction decoder 106. Theinstruction decoder 106 may control other elements, as well, to providethe missing tensor. For example, instruction decoder 106 could send acontrol signal to multiplexer 230 to select either a tensor from tensoroperation component 102(1) or tensor register file 110(5). Otherpossibilities exist for providing the missing tensor.

In the event that two tensor operands are needed for the subject machineinstruction, then steps 514-518 are performed. Step 514 may includecircuitry in the hardware accelerator identifying a machine instructionthat is to provide the missing tensor operand in a similar manner asstep 510. The identification of the other machine instruction in step514 may be similar to step 510.

Step 516 includes using the resultant tensor from the machineinstruction identified in step 514 as a first tensor operand for thesubject machine instruction. In one embodiment, the crossbar component236 in tensor operation component 102(2) supplies the first (missing)tensor to tensor operation calculator 112(2) or to tensor operationcalculator 112(3). Note that the missing tensor operand might come fromanother tensor operation calculator 112 in tensor operation component102(2). The crossbar component 236 may act in response to a controlsignal from the instruction decoder 106. The instruction decoder 106 maycontrol other elements, as well, to provide the first (missing) tensor.For example, instruction decoder 106 could send a control signal tomultiplexer 230 to select either a tensor from tensor operationcomponent 102(1) or tensor register file 110(5). Other possibilitiesexist for providing the missing tensor.

Step 518 includes using a tensor that is at least partially specified inthe subject machine instruction as a second tensor operand for thesubject machine instruction. For example, a tensor from tensor registerfile 110(2) may be provided to tensor operation calculator 112(2). Asanother example, a tensor from tensor register file 110(3) may beprovided to tensor operation calculator 112(3).

Note that in some embodiments, the second tensor is only partiallyspecified in the subject machine instruction. In one embodiment, thesubject machine instruction does not expressly specify which of thetensor register files 110 contains the second tensor. Hence, in oneembodiment, the subject machine instruction “only partially specifies”the second tensor operand. In one embodiment, the hardware acceleratorinfers which tensor register file 110 contains the second tensor basedon the type of tensor operation to be performed. For example, if thesubject machine instruction is a vector-vector multiply, the hardwareaccelerator 100 may infer that the second tensor is located in a tensorregister file 110 that is dedicated to a tensor operation calculator 112that is configured to perform vector-vector multiplication. The subjectmachine instruction may expressly specify a location within a tensorregister file, however. For example, the subject machine instruction mayexpressly contain an index value. Note that the subject machineinstruction could completely specify the second tensor operand. Forexample, the subject machine instruction could also specify a tensorregister file that contains the second tensor operand.

FIGS. 6A-6D are flowcharts of embodiments of processes of differenttechniques that may be used to use a tensor that results from onemachine instruction as a missing tensor operand in another machineinstruction.

FIG. 6A is a flowchart of one embodiment of a process 600 of executingmachine instructions in a hardware accelerator 100. Process 600 is foran example in which the machine instruction having the missing operandis for a single tensor operation. By a “single tensor operation” it ismeant that the operation involves just one tensor.

In step 602, a first machine instruction is executed in a first tensoroperation calculator 112 to generate a tensor. Herein, this is referredto as a “resultant tensor.” This might be any of the tensor operationcalculators 112 in FIG. 1 or 2, but is not limited to those examples.The first tensor operation calculator 112 might be configured to performeither a single tensor operation or a two tensor operation. Step 602 isone embodiment of step 406 in process 400.

For the sake of illustration, in step 602, a matrix-vector multiplymight be performed in tensor operation calculator 112(1), avector-vector multiply might be performed in one of the instances oftensor operation calculator 112(2), vector-vector addition (orsubtraction) might be performed in one of the instances of tensoroperation calculator 112(3), or a single tensor operation might beperformed in one of the instances of tensor operation calculator 112(4).In one embodiment, each of these tensor operations results in a vector.

In step 604, the resultant tensor is routed from the first tensoroperation calculator 112 to an input of a second tensor operationcalculator 112. In one embodiment, this resultant tensor is routedwithout intermediate storage of the resultant tensor in a register orother storage. In one embodiment, instruction decoder 106 issues controlsignals to route the resultant tensor.

In step 606, a second machine instruction is executed in the secondtensor operation calculator 112 based on the resultant tensor. As noted,the second tensor operation calculator 112 is configured to perform asingle tensor operation. With respect to the example of FIG. 2, theresultant tensor may be routed to either tensor operation calculator112(4) of tensor operation component 102(2) or tensor operationcalculator 112(4) of tensor operation component 102(3). Step 606 is oneembodiment of step 408 in process 400.

Using the example of tensor operation calculator 112(4) of tensoroperation component 102(2) receiving the resultant tensor, the resultanttensor might be provided directly from any of (but not limited to)tensor operation calculator 112(2) of tensor operation component 102(2),tensor operation calculator 112(3) of tensor operation component 102(2),or tensor operation calculator 112(1) of tensor operation component102(1). In one embodiment, instruction decoder 106 provides controlsignals to one or more of tensor operation component 102(1), multiplexer230, and/or tensor operation component 102(2) to route the resultanttensor. The instruction decoder 106 could issue control signals to otherelements in the hardware accelerator 100, as well, to route theresultant tensor.

FIG. 6B is a flowchart of one embodiment of a process 620 of executingmachine instructions in a hardware accelerator 100. Process 620 is foran example in which the machine instruction having the missing operandis for a two tensor operation.

In step 622, a first machine instruction is executed in a first tensoroperation calculator 112 to generate a tensor. This might be any of thetensor operation calculators 112 in FIG. 1 or 2, but is not limited tothose examples. The first tensor operation calculator 112 might beconfigured to perform either a single tensor operation or a two tensoroperation. Step 622 is one embodiment of step 406 in process 400.

For the sake of illustration, in step 622, a matrix-vector multiplymight be performed in tensor operation calculator 112(1) to generate avector, a vector-vector multiply might be performed in one of theinstances of tensor operation calculator 112(2), vector-vector addition(or subtraction) might be performed in one of the instances of tensoroperation calculator 112(3), or a single tensor operation might beperformed in one of the instances of tensor operation calculator 112(4).

In step 624, the resultant tensor is routed from the first tensoroperation calculator 112 to a first tensor input of a second tensoroperation calculator 112. In one embodiment, this resultant tensor isrouted without intermediate storage of the resultant tensor in aregister or other storage. In one embodiment, instruction decoder 106issues control signals to route the resultant tensor. With respect tothe example of FIG. 2, the resultant tensor may be routed to tensoroperation calculator 112(2) or tensor operation calculator 112(3) ineither tensor operation component 102(2) or 102(3). In one embodiment,the resultant tensor is routed through crossbar 236.

In step 626, a second machine instruction is executed in the secondtensor operation calculator 112 based on the routed tensor and a secondtensor from a tensor register file. With respect to the example of FIG.2, the second tensor may come from tensor register file 110(2) whentensor operation calculator 112(2) is the second tensor operationcalculator. The second tensor may come from tensor register file 110(3)when tensor operation calculator 112(3) is the second tensor operationcalculator. Step 626 is one embodiment of step 408 of process 400. Inone embodiment, the second tensor is only partially specified by thesecond machine instruction, as discussed with respect to step 518 inprocess 500.

The first machine instruction (which provides the missing tensoroperand) need not be executed in a tensor operation calculator 112. FIG.6C is a flowchart of one embodiment of a process 640 in which the firstmachine instruction is a tensor load instruction. In step 642, a firstmachine instruction is executed to load a tensor into a tensor registerfile 110. In one embodiment, the first machine instruction indicates amemory location external to the hardware accelerator 100 from which thetensor is to be accessed. Step 642 is one embodiment of step 406 ofprocess 400. This tensor load machine instruction is at or at least nearthe beginning of the chain of instructions, in one embodiment. In oneembodiment, this tensor load machine instruction comes prior to anymachine instruction to perform a tensor calculation in, for example, oneof the tensor operation calculator 112. Thus, the first machineinstruction may load an “initial tensor” for the chain of machineinstructions.

In one embodiment, a tensor is loaded into one or more of the tensorregister files 110(1)-110(5) of FIG. 2, in step 642. In one embodiment,the first machine instruction does not expressly specify which tensorregister file to load the tensor. The hardware accelerator 100determines which of the tensor register files should receive the tensorbased on the type of tensor operation calculator 112 that might use thetensor, in one embodiment. Further details are discussed below.

Step 644 includes executing the second machine instruction in a tensoroperation calculator 112 using the initial tensor that was loaded intothe tensor register file 110. Step 664 is one embodiment of step 408 ofprocess 400. In one embodiment, the tensor operation calculator 112 alsouses a second tensor. As one example, step 644 may include tensoroperation calculator 112(1) using an initial tensor loaded into tensorregister file 110(4) in tensor operation component 102(1) by the firstmachine instruction, and a matrix from tensor register file 110(1). Asanother example, step 644 may include tensor operation calculator 112(2)using an initial tensor loaded into tensor register file 110(5) inmulti-function initial tensor component 234 by the first machineinstruction, and a tensor from tensor register file 110(2). As anotherexample, step 644 may include tensor operation calculator 112(4) usingan initial tensor loaded into tensor register file 110(5) inmulti-function initial tensor component 234 by the first machineinstruction in a single tensor operation. Many other possibilities existfor step 644.

Note that the second machine instruction does not necessarily involveperforming a tensor calculation in a tensor operation calculator 112.For example, the second machine instruction may be an instruction tostore a tensor to memory external to the hardware accelerator 100. Inthis case, the second machine instruction may come at the end of thechain of instructions.

FIG. 6D is a flowchart of one embodiment of a process 660 in which thesecond machine instruction is a tensor store instruction. In step 662, afirst machine instruction is executed in a tensor operation calculator112 to generate a tensor. This might be a single tensor operation or atwo tensor operation. Thus, this could involve any of the tensoroperation calculators 112(1)-112(4) in FIG. 2, for example. Step 662 isone embodiment of step 406 of process 400.

Step 664 includes executing a second machine instruction to store theresultant tensor from the first machine instruction. As one example, theinstruction decoder 106 might issue control signals to tensor operationcomponent 102(3) and to de-multiplexer 232 to route a tensor from one ofthe tensor operation calculators 112 in tensor operation component102(3) to the output queue 222. The tensor may then be transferred fromoutput queue 222 to memory external to the hardware accelerator 100.Step 664 is one embodiment of step 408 of process 400.

FIG. 7 is a flowchart of one embodiment of a process 700 of accessing atensor from a tensor register file 110. The process 700 involvesexecuting a machine instruction 302 that needs a tensor from a tensorregister file 110. In process 700, the machine instruction is notrequired to specify what tensor register file 110 contains the tensor.In one embodiment, the hardware accelerator 100 infers what tensorregister file 110 to access based on the type of tensor operation thatis to be performed.

Step 702 includes the instruction decoder 106 decoding an opcode of amachine instruction to determine what type of tensor operation is to beperformed to execute the machine instruction. In step 702 theinstruction decoder 106 may also determine an index into a tensorregister file 110. In one embodiment, the machine instruction has anoperand field that specifies the index into a tensor register file 110.However, the actual tensor register file is not specified in the machineinstruction of one embodiment.

Referring to FIG. 3A, machine instructions 302(2) and 302(3) are twoexamples of machine instructions that have an operand field thatspecifies an index into a tensor register file 110, without specifyingan actual tensor register file. In FIG. 3A, the contents between theparentheses is the operand field. Recall that the “XXX” is merely torepresent that the tensor operand is missing. That is, XXX means thatthe machine instruction 302 does not have any bits to specify a tensoroperand that is needed to execute the machine instruction. The “Wx” inmachine instruction 302(2) may be a one byte value that is an index intoa tensor register file 110, as one example. The “b” in machineinstruction 302(3) may be a one byte value that is an index into atensor register file 110, as one example.

Step 704 includes selecting a tensor operation calculator 112 to performthe tensor operation. In one embodiment, the tensor operation calculator112 has a tensor register file 110 that is dedicated to the tensoroperation calculator 112. That is, the tensor operation calculator 112is the only component that accesses this instance of the tensor registerfile 110, in one embodiment. Step 704 may be performed by instructiondecoder 106 and/or crossbar 236, but is not limited to those elements.

With reference to the example machine instruction 302(2) formatrix_vector multiply, tensor operation calculator 112(1) might beselected. In this example, tensor operation calculator 112(1) isconfigured to perform matrix-vector multiplies. With reference to theexample machine instruction 302(3) for vector_vector add, tensoroperation calculator 112(3) might be selected. In this example, tensoroperation calculator 112(3) is configured to perform vector-vectoraddition.

Step 706 includes accessing a tensor from a tensor register file that isdedicated to the selected tensor operation calculator 112. The indexfrom the machine instruction 302 may be used to determine the locationof the tensor within the tensor register file.

With reference to the example in which tensor operation calculator112(1) is selected, tensor register file 110(1) may be accessed. In thisexample, the index “Wx” may be used to select an entry in tensorregister file 110(1) to provide a matrix to tensor operation calculator112(1).

With reference to the example in which tensor operation calculator112(3) is selected, tensor register file 110(3) may be accessed. In thisexample, the index “b” may be used to select an entry in tensor registerfile 110(3) to provide a tensor to tensor operation calculator 112(3).

Note that an outcome of step 706 is that the correct tensor registerfile 110 is accessed without the machine instruction specifying which ofthe tensor register files 110 to access. This may be referred to hereinas implied tensor register file read access.

Step 708 includes the selected tensor operation calculator 112performing the tensor operation using the tensor that was accessed fromthe tensor register file 110. The tensor operation calculator 112 mayalso input another tensor in order to perform a tensor operation, suchas, but not limited to, matrix-matrix multiply, matrix-vector multiply,vector-vector multiply, vector-vector addition, or vector-vectorsubtraction.

As noted above, rather than having a large tensor register file thatstores tensors for all types of tensor operations, some embodiments ofthe hardware accelerator 100 have multiple smaller tensor register files110, at least some of which are used to store tensors for a specific setof one or more types of tensor operations. Each smaller tensor registerfile 110 may be dedicated to one instance of a tensor operationcalculator 112. Such a configuration may improve bandwidth by keepingthe various tensor operation calculators 112 busy a high percentage ofthe time. For example, having the dedicated tensor register files 110may avoid or at least reduce stalls that could otherwise occur while atensor operation calculator 112 waits for a tensor to be provided fromstorage.

FIG. 8 is a flowchart of one embodiment of a process 800 of storingtensors in tensor register files 110. The process 800 may be implementedin a hardware accelerator 100 such as those embodiments depicted inFIGS. 1 and 2, but is not limited to those embodiments. The hardwareaccelerator 100 may also be referred to as a tensor processor. In oneembodiment, the tensor processor has a number of tensor operationcalculators 112 and a number of tensor register files 110. Each of thetensor operation calculators 112 may be configured to perform a type oftensor operation. Each of the tensor register files 110 may beassociated with one of the tensor operation calculators 112. In oneembodiment, each of the tensor register files 110 is dedicated to one ofthe tensor operation calculators 112. Each instance of a tensoroperation calculator may be configured to perform a set of one or moretypes of vector operations. Moreover, there may be variation infunctionally between the tensor operation calculators, such thatdifferent tensor operation calculators perform different sets of typesof vector operations. This configuration may, in effect, result indifferent tensor register files 110 that are associated with differentsets of one or more types of tensor operations.

Step 802 includes determining a type of tensor operation that the tensorprocessor is to perform on a tensor. For example, circuitry in thetensor processor may determine whether this tensor is to be used for avector-vector multiply, a vector-vector addition, a matrix-vectormultiply, a vector-vector division, etc. In one embodiment, instructiondecoder 106 makes this determination based on a value of a filed in amachine instruction to store the tensor in the tensor processor.

Step 804 includes storing the tensor in one or more instances of atensor register file 110. In one embodiment, the tensor is stored in atensor register file associated with a tensor operation calculator thatis configured to perform the type of tensor operation. For example, ifthe tensor is to be used for matrix-vector multiply, then the tensor maybe stored in a tensor register file 110 that is associated with a tensoroperation calculator 112 that is configured to perform matrix-vectormultiply. In one embodiment, the tensor is stored in all instances oftensor register files 110 that are associated with a tensor operationcalculator 112 that is configured to perform matrix-vector multiply.

In one embodiment, writing to tensor register files 110 is controlledbased on machine instructions. In one embodiment, a value in a field ina machine instruction to store a tensor indicates the type of tensorregister file 110 into which the tensor should be stored. FIG. 9 is aflowchart of one embodiment of a process 900 of storing tensors in atensor processor in response to a machine instruction 302 to store atensor in a tensor processor 100.

Step 902 includes decoding a machine instruction to store a tensor inthe tensor processor. Step 902 includes accessing a value in an operandfield of the machine instruction, in one embodiment.

Step 904 includes determining a type of tensor register file 110 inwhich to store the tensor, based on the decoded machine instruction. Thevalue in the operand field indicates the type of tensor register file110 into which the tensor should be loaded, in one embodiment. As notedabove, in some embodiments, each tensor register file 110 is associatedwith a set of one or more types of tensor operations. This may be basedon what tensor operator calculator 112 the tensor register file 110 isdedicated to. Moreover, the tensor register files 110 can be classifiedinto different types, based on the types of tensor operations associatedwith the tensor register files 110. In one embodiment, the value in theoperand field is used to select one of the types of tensor registerfiles 110.

For example, the value may indicate whether the tensor is to be storedin a tensor register file 110 that is used to store matrices formatrix-vector multiplies, a tensor register file 110 that is used tostore tensors for matrix-vector multiplies, a tensor register file 110that is used to store tensors for vector-vector multiplies, a tensorregister file 110 that is used to store vectors for vector-vectoraddition/subtraction, etc. Note that a tensor register file 110 may beused to store matrices for matrix-vector multiplies if it is dedicatedto a tensor operator calculator 112 configured to perform matrix-vectormultiplies, etc.

Step 906 includes writing the tensor to all instances of the type oftensor register file 110. For some types of tensor register files 110,there may be only one instance the tensor register file 110.

In one embodiment, reading from tensor register files 110 is controlledbased on a type of tensor operation to be performed to execute a machineinstruction. For example, a tensor register file 110 that storesmatrices for matrix-vector multiplies might only be accessed in responseto executing a machine instruction to perform a matrix-vector multiply.As another example, a tensor register file 110 that stores tensors formatrix-vector multiplies might only be accessed in response to executinga machine instruction to perform a matrix-vector multiply. As stillanother example, a tensor register file 110 that stores tensors forvector-vector multiplies might only be accessed in response to executinga machine instruction to perform a vector-vector multiply. As a furtherexample, tensor register file 110 that stores tensors for vector-vectoraddition/subtraction might only be accessed in response to executing amachine instruction to perform a vector-vector addition/subtraction.

In one embodiment, the hardware accelerator 100 controls access to thetensor register files 110 based on a type of tensor operation that amachine instruction is to perform when executed in one of the tensoroperation calculators 112. FIG. 10 is a flowchart of one embodiment of aprocess 1000 of accessing tensors from tensor register files 110 in atensor processor in which access to the tensor register files 110 iscontrolled based on a type of tensor operation of a machine instruction.In the examples of process 1000, three different tensor operations areconsidered (vector-vector multiply, vector-vector add/subtract,matrix-vector multiply). The various types of tensor operations in theprocess 1000 are for purpose of illustration. The process 1000 can bemodified to include many other types of tensor operations, or to dropsome of those depicted in FIG. 10.

In step 1002, a machine instruction is decoded. In one embodiment, theinstruction decoder 106 performs step 1002. The instruction decoded 106may determine what type of tensor instruction is to be performed basedon an operation code of the machine instruction.

If the machine instruction is for a vector-vector multiply then steps1006-1008 may be performed. In step 1006, circuitry in the tensorprocessor determines an instance of a vector multiply calculator that isto perform the tensor operation. For the sake of discussion, tensoroperation calculator 112(2) in tensor operation component 102(2) isselected. In step 1008, a tensor is accessed from the instance of thetensor register file adjacent to the selected tensor operationcalculator 112(2). Referring to FIG. 2 for example, a tensor is accessedfrom tensor register file 110(2) in tensor operation component 102(2).Note that the tensor operation calculator 112(2) may be provided withanother tensor from elsewhere than tensor register file 110(2) for theother tensor for the vector-vector multiply.

If the machine instruction is for a vector-vector addition/subtractionthen steps 1010-1012 may be performed. Note that in this case, steps1010-1012 may be performed whether the tensor operation is addition orsubtraction. In step 1010, circuitry in the tensor processor determinesan instance of a vector addition/subtraction calculator that is toperform the tensor operation. For the sake of discussion, tensoroperation calculator 112(3) in tensor operation component 102(2) isselected. In step 1012, a tensor is accessed from the instance of thetensor register file adjacent to the selected tensor operationcalculator 112(3). Referring to FIG. 2 for example, a tensor is accessedfrom tensor register file 110(3) in tensor operation component 102(2).Note that the tensor operation calculator 112(2) may be provided withanother tensor from elsewhere than tensor register file 110(2) for theother tensor for the vector-vector addition/subtraction.

Note that one of several different types of opcodes may be used tospecify the vector-vector addition/subtraction. For example, oneoperation code may be used to specify vector-vector addition, anotheroperation code may be used to specify vector-vector subtraction in whichthe tensor in tensor register file 110(3) is the minuend, anotheroperation code may be used to specify vector-vector subtraction in whichthe tensor in tensor register file 110(3) is the subtrahend.

If the machine instruction is for a matrix-vector multiply then steps1014-1016 may be performed. In step 1014, circuitry in the tensorprocessor determines an instance of a matrix-vector multiply calculatorthat is to perform the tensor operation. For the sake of discussion,tensor operation calculator 112(1) in tensor operation component 102(1)is selected. In step 1016, a tensor is accessed from the instance of thetensor register file adjacent to the selected tensor operationcalculator 112(2). Referring to FIG. 2 for example, a tensor is accessedfrom tensor register file 110(4) in tensor operation component 102(1).As another example, a matrix is accessed from tensor register file110(1) in tensor operation component 102(1).

Some embodiments of a hardware accelerator have a native size of tensorsin a machine instruction. For example, a matrix-vector multiply mayassume a 100 by 100 matrix, and a vector of 100 elements. In oneembodiment, a tiling factor may be specified in one of the machineinstructions to allow a tensor having other than the native size to beprocessed. The tiling factor may be used to adjust the number of rowsand/or columns of a matrix. For example, a tiling factor of two for rowsand three for columns indicates that the matrix-vector multiplyinstruction is to operate on a 200 by 300 matrix. Vectors may also havea native size that may be scaled by the tiling factor. For example, avector may have a native size of 100 elements.

The tiling factor allows the tensor processor to operate moreefficiently. For example, instruction throughput may be improved. Notethat instruction execution rate in the tensor processor can be extremelyfast. Hence, instruction decoding and issuance to the tensor operationcomponents 102 needs to be fast to prevent a bottleneck. The tilingfactor helps to keep instruction decoding/issuance from falling behindinstruction execution. Additionally, the tiling factor may simplify thecode (e.g., the chain of machine instructions).

The following two code snippets show how one embodiment of a tilingfactor may be used to use significantly fewer machine instructions.

TABLE I  1. for (int r = 0; r < rows; r++) {  2.  for (int c = 0; c <cols; c++) {  3. v_rd(InitialVRF, v1 + c);  4. mv_mul(m1 + cols * r +c);  5. if (c != 0) w_add(v2);  6.  if (c != cols−1 ) v_wr(AddSubVRF,v2);  7. }  8. vv_add(v3 + r);  9. v_sigmO; 10. v_wr(NetOutputQ, v4 +r); 11.  }

TABLE II 1. setLogicalRowsCols(rows, cols) 2 v_rd(InitialVRF, v1); 3.mv_mul(m1); 4. w_add(v3); 5. v_sigmO; 6. v_wr(NetOutputQ, v4);

The code snippet in each table may perform the same functionality, butthe code in the Table II has fewer instructions. The tiling factor maybe set by “setLogicalRowsCols”. FIG. 11 is a diagram that illustratesfunctionality that may be achieved by executing the code in Table II inone embodiment of a hardware accelerator 100. The tiling factor has beenset to two rows and three columns in the example of FIG. 11. Whendiscussing FIG. 11, reference will be made to various elements in FIG. 2for sake of illustration.

Referring to FIG. 11, machine instruction “v_rd(InitialVRF, v1)” may beexecuted multiple times (1102(0)-1102(5)) across three differentaddresses (v1+0, v1+1, v1+2). Each address includes a base address(e.g., “v1”) and an offset. The offset may be based on the native tensorsize. This instruction reads in tensors to a tensor register file 110such as tensor register file 110(4) in tensor operation component102(1). Note that the native size of a tensor in which all of theelements are scalar elements might be 100 scalar elements, as oneexample. Thus, the net result is that the tensor to be loaded has 300scalar elements, as one example.

Machine instruction “mv_mul(m1)” may be executed six times(1104(0)-1104(5)) across six different addresses (m1+0, m1+1, m1+2,m1+3, m1+4, m1+5). This instruction performs matrix-vector multipliesin, for example, tensor operation calculator 112(1). The matrices areidentified based on offsets from the address “m1” in this instruction.The tensors for the matrix-vector multiplies are the ones loaded by theprevious v_rd(InitialVRF, v1) instruction.

Machine instruction “w_add(v3)” is executed across two addresses (v3+0,v3+1). These addresses are based on offsets from the “v3” in thisinstruction. Thus, in effect, two unique vectors of the “native size”are specified. This instruction performs vector addition in, forexample, tensor operation calculator 112(3) in in tensor operationcomponent 102(2). The vectors specified by v3+0, v3+1 may come fromtensor register file 110(3), for example. The other vectors for theaddition may be resultant tensors from the matrix-vector multiply.

Machine instruction “v_sigm( )” may be executed twice (1108(0),1108(1)). This may take the sigmoid of each element in the resultanttensors of the vector-vector addition. The result of this instructionmay be stored in two “native size” vectors in a tensor register file110.

Machine instruction “v_wr(NetOutputQ, v4)” may be executed twice(1110(0), 1110(1)) across two addresses (v4+0, v4+1). This instructionmay transfer two “native size” vectors from the tensor register file 110holding the result of the sigmoid into the output queue 222.

In one embodiment, the tiling factor is used on a chain by chain basis.FIG. 12 is a flowchart of one embodiment of a process 1200 of executinga chain of machine instructions in a tensor processor based on a tilingfactor. Step 1202 includes a determination of whether a new chain hasstarted. Step 1202 may be similar to step 352 in FIG. 3B.

Step 1204 is a determination of whether a tiling factor has beenspecified for this chain. In one embodiment, the tiling factor isspecified by an instruction in the chain.

If there is not a tiling factor, then the next instruction is the chainis executed using a native size of tensors in the tensor processor 100,in step 1206. The tensor processor continues to execute additionalinstructions until an end of the chain is detected, in step 1208. Theend of the chain may be detected as in step 358 of FIG. 3B.

If there is a tiling factor, then the next instruction is the chain isexecuted while scaling the native size of tensors in the tensorprocessor, in step 1210. The tensor processor continues to executeadditional instructions until an end of the chain is detected, in step1212.

FIG. 13 illustrates an example environment 1300 in which an embodimentof a hardware accelerator 100 as described herein can operate. In someembodiments, the example environment 1300 can be used to execute machinelearning algorithms, such as deep neural networks. In some examples, thevarious devices and/or components of environment 1300 include a varietyof computing devices 1302. By way of example and not limitation,computing devices 1302 may include devices 1302 a-1302 e. Althoughillustrated as a diverse variety of device types, computing devices 1302can be other device types and are not limited to the illustrated devicetypes. In some implementations any of a number of computing devices 1302may be interconnected via a network 1304.

Network 1304 can include, but is not limited to, a cellular network(e.g., wireless phone), a point-to-point dial up connection, a satellitenetwork, the Internet, a local area network, a wide area network, a WiFinetwork, an ad hoc network, an intranet, an extranet, or a combinationthereof. Network 1304 may include one or more connected networks (e.g.,a multi-network environment). Network 1304 may include one or more datacenters that store and/or process information (e.g., data) received fromand/or transmitted to computing devices 1302.

In an implementation, computing devices 1302 can include any type ofdevice with one or multiple processors 1306 operably connected to aninput/output interface 1308, a hardware accelerator 100, and a memory1312, e.g., via a bus 1314. Computing devices 1302 can include personalcomputers such as, for example, desktop computers 1302 a, laptopcomputers 1302 b, tablet computers 1302 c, data center servers 1302 d(or servers is any other environment), smart phones 1302 e, electronicbook readers, wearable computers, automotive computers, gaming devices,etc. In an implementation, computing devices 1302 need not includeprocessor 1306, and may be a hardware appliance.

Computing devices 1302 also can include other computing devices such as,for example, server computers, thin clients, terminals, and/or workstations. In some examples, computing devices 1302 can include, forexample, components for integration in a computing device, appliances,or other sorts of devices.

In some examples, some or all of the functionality described as beingperformed by computing devices 1302 may be implemented by one or moreremote peer computing devices, a remote server or servers, or a cloudcomputing resource. In some examples, a computing device 1302 mayinclude an input port to receive an input data sequence. Computingdevice 1302 may further include one or multiple processors 1306 toperform machine learning processing, for example.

In some examples, as shown regarding device 1302 d, memory 1312 canstore instructions executable by the processor(s) 1306 including anoperating system 1316, and programs or applications 1320 that areloadable and executable by processor(s) 1306. Applications 1320 mayinclude machine learning processing applications 1320 that may beexecuted to operate hardware accelerator 100, for example. The one ormore processors 1306 may include one or more central processing units(CPUs), graphics processing units (GPUs), video buffer processors, andso on.

In some implementations, machine learning processing applications 1320include executable code stored in memory 1312 and executable byprocessor(s) 1306 to receive and implement machine learning algorithmsthat include data sequences (e.g., streaming data or data files),locally or remotely by computing device 1302, via input/output 1308. Insome examples, the data sequences may be associated with one or moreapplications 1320. Machine learning processing applications 1320 mayoperate in combination with hardware accelerator 100 to apply any of anumber of processes used to process data stored in memory 1312 orreceived via input/output 1308.

Although certain blocks have been described as performing variousoperations, the modules are merely examples and the same or similarfunctionality may be performed by a greater or lesser number of modules.Moreover, the functions performed by the modules depicted need notnecessarily be performed locally by a single device. Rather, someoperations could be performed by a remote device (e.g., peer, server,cloud, etc.).

Alternatively, or in addition, some or all of the functionalitydescribed herein can be performed, at least in part, by one or morehardware logic circuits. For example, and without limitation,illustrative types of hardware logic circuits that can be used includean FPGA device, an application-specific integrated circuit (ASIC)device, a GPU, a massively parallel processor array (MPPA) device, anapplication-specific standard product (ASSP) device, a system-on-a-chipdevice (SOC) device, a complex programmable logic device (CPLD), acustom integrated circuit, etc.

Computer readable media may include computer storage media and/orcommunication media. Computer storage media includes volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules, or other data.Computer storage media includes, but is not limited to, phase changememory (PRAM), static random-access memory (SRAM), dynamic random-accessmemory (DRAM), other types of random-access memory (RAM), read-onlymemory (ROM), electrically erasable programmable read-only memory(EEPROM), flash memory or other memory technology, compact diskread-only memory (CD-ROM), digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other non-transmissionmedium that can be used to store information for access by a computingdevice.

In contrast, communication media embodies computer readableinstructions, data structures, program modules, or other data in amodulated data signal, such as a carrier wave, or other transmissionmechanism. As defined herein, computer storage media does not includecommunication media. In various examples, memory 1312 is an example ofcomputer storage media storing computer-executable instructions.

In various examples, an input device of input/output interface 1308 canbe a direct-touch input device (e.g., a touch screen), an indirect-touchdevice (e.g., a touch pad), an indirect input device (e.g., a mouse,keyboard, a camera or camera array, etc.), or another type ofnon-tactile device, such as an audio input device.

Computing device(s) 1302 also may include one or more input/outputinterfaces 1308 to allow computing device 1302 to communicate with otherdevices. Input/output interface 1308 can include one or more networkinterfaces to enable communications between computing device 1302 andother networked devices such as other device(s) 1302. Input/outputinterface 1308 can allow a computing device 1302 to communicate withother devices such as user input peripheral devices (e.g., a keyboard, amouse, a pen, a game controller, a voice input device, a touch inputdevice, gestural input device, and the like) and/or output peripheraldevices (e.g., a display, a printer, audio speakers, a haptic output,and the like).

One embodiment includes a method of operating a tensor processor havinga plurality of tensor operation calculators each configured to perform atype of tensor operation and plurality of tensor register files. Each ofthe tensor register files is dedicated to one of the plurality of tensoroperation calculators. The method comprises: determining a type oftensor operation that the tensor processor is to perform on respectiveones of a plurality of tensors; and storing each respective tensor ofthe plurality of tensors in a tensor register file dedicated to a tensoroperation calculator that is configured to perform the type of tensoroperation the tensor processor is to perform on the respective tensor.

One embodiment includes a tensor processor, comprising a plurality oftensor operation calculators each configured to perform a type of tensoroperation from a plurality of types of tensor operations and a pluralityof tensor register files. Each of the tensor register files is dedicatedto one of the plurality of tensor operation calculators. The tensorprocessor further comprises circuitry configured to determine a type oftensor operation that the tensor processor is to perform on respectiveones of a plurality of tensors. The tensor processor further comprisescircuitry configured to store each respective tensor in a tensorregister file dedicated to a tensor operation calculator that isconfigured to perform the type of tensor operation the tensor processoris to perform on the respective tensor.

One embodiment includes a method of executing machine instructions in atensor processor. The method comprises accessing a first machineinstruction in a chain of machine instructions, and accessing a secondmachine instruction in the chain of machine instructions. The secondmachine instruction is missing a tensor operand needed to execute thesecond machine instruction. The method also comprises executing thefirst machine instruction in the tensor processor to result in aresultant tensor. The method also comprises executing the second machineinstruction in the tensor processor using the resultant tensor as themissing tensor operand.

One embodiment includes a tensor processor comprising: circuitryconfigured to decode a subject machine instruction in a chain of machineinstructions; circuitry configured to determine that the subject machineinstruction is missing a tensor operand needed to execute the subjectmachine instruction; circuitry configured to identify another machineinstruction in the chain that is to provide the missing tensor operand;circuitry configured to execute the other machine instruction to resultin a resultant tensor; and circuitry configured to execute the subjectmachine instruction using the resultant tensor as the missing tensoroperand.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. An apparatus comprising: a plurality of tensoroperation calculators each configured to perform a type of tensoroperation of a plurality of types of tensor operations, the plurality oftensor operation calculators comprising multiple instances of tensoroperation calculators configured to perform a first type of the tensoroperations; a plurality of tensor register files, each of the tensorregister files being associated with one of the plurality of tensoroperation calculators; and logic configured to store tensors in theplurality of tensor register files in accordance with the type of tensoroperation to be performed on the respective tensors; wherein the logicis further configured to store multiple separate copies of tensors, forwhich the apparatus is to perform the first type of tensor operation, ineach of the dedicated tensor register files that are associated with themultiple instances of the tensor operation calculators configured toperform the first type of tensor operation, to the exclusion of othersof the plurality of tensor register files.
 2. The apparatus of claim 1,wherein to store respective ones of the tensors in the tensor registerfiles in accordance with the type of tensor operation the logic isfurther configured to: determine the type of tensor operation that theapparatus is to perform on respective ones of the tensors; and store therespective tensors into tensor register files that are dedicated to atensor operation calculator that is configured to perform the type oftensor operation.
 3. The apparatus of claim 2, wherein to determine thetype of tensor operation the logic is further configured to: access anidentifier in a machine instruction to load a tensor, wherein theidentifier is associated with a type of the tensor operations.
 4. Theapparatus of claim 1, wherein the logic is further configured to:control read access to the plurality of tensor register files based on atype of tensor operation that a machine instruction is to perform whenexecuting in one of the tensor operation calculators.
 5. The apparatusof claim 1, wherein a first of the tensor operation calculators isconfigured to perform a tensor-tensor operation, wherein the logic isfurther configured to: provide a tensor from the tensor register fileassociated with the first tensor operation calculator as an operand tothe first tensor operation calculator.
 6. The apparatus of claim 1,wherein a first of the tensor operation calculators is configured toperform a matrix-vector operation, wherein the logic is furtherconfigured to: provide a matrix from the tensor register file associatedwith the first tensor operation calculator as an operand to the firsttensor operation calculator.
 7. The apparatus of claim 1, wherein afirst of the tensor operation calculators is configured to perform avector-vector operation, wherein the logic is further configured to:provide a vector from the tensor register file associated with the firsttensor operation calculator as an operand to the first tensor operationcalculator.
 8. A method of operating a tensor processor having aplurality of tensor operation calculators each configured to perform atype of tensor operation, the plurality of tensor operation calculatorscomprising multiple instances of tensor operation calculators configuredto perform a first type of tensor operation, and a plurality of tensorregister files, each of the tensor register files being dedicated to oneof the plurality of tensor operation calculators, the method comprising:determining a type of tensor operation that the tensor processor is toperform on respective ones of a plurality of tensors; and storing eachrespective tensor of the plurality of tensors in a tensor register filededicated to a tensor operation calculator that is configured to performthe type of tensor operation the tensor processor is to perform on therespective tensor; wherein the storing comprises storing multipleseparate copies of tensors, for which the tensor processor is to performthe first type of tensor operation, in each of the dedicated tensorregister files that are associated with the multiple instances of thetensor operation calculators configured to perform the first type oftensor operation, to the exclusion of others of the plurality of tensorregister files.
 9. The method of claim 8, further comprising: decodingmachine instructions to perform respective ones of the types of tensoroperations; and controlling read access to the plurality of tensorregister files based on the type of tensor operation that a machineinstruction is to perform when executed in one of the tensor operationcalculators.
 10. The method of claim 9, wherein controlling read accessto the plurality of tensor register files based on the type of tensoroperation that a machine instruction is to perform when executed in oneof the tensor operation calculators comprises: only allowing read accessto a first of the tensor register files that is dedicated to a first ofthe tensor operation calculators responsive to executing a first type oftensor operation in the first tensor operation calculator; and onlyallowing read access to a second of the tensor register files that isdedicated to a second of the tensor operation calculators responsive toexecuting a second type of tensor operation in the second tensoroperation calculator.
 11. The method of claim 8, wherein determining thetype of tensor operation that the tensor processor is to perform on therespective ones of the plurality of tensors comprises: accessing anidentifier of a type of tensor register file from a machine instructionto load a tensor, wherein the identifier is associated with a type oftensor operation.
 12. The method of claim 8, wherein the storing furthercomprises: storing a matrix in a tensor register file dedicated to amatrix-vector operation calculator; and storing a vector in a tensorregister file dedicated to a vector-vector operation calculator.
 13. Themethod of claim 8, further comprising: decoding a machine instruction todetermine a type of tensor operation to be performed in the tensorprocessor; determining an instance of a tensor operation calculator inthe tensor processor to execute the machine instruction; accessing atensor from a tensor register file that is dedicated to the instance ofthe tensor operation calculator; and executing the type of tensoroperation in the tensor operation calculator using the accessed tensor.14. A tensor processor, comprising: a plurality of tensor operationcalculators each configured to perform a type of tensor operation from aplurality of types of tensor operations, the plurality of tensoroperation calculators comprising multiple instances of tensor operationcalculators configured to perform a first type of tensor operation fromthe plurality of types of tensor operations; a plurality of tensorregister files, each of the tensor register files being dedicated to oneof the plurality of tensor operation calculators; a first circuitryconfigured to determine a type of tensor operation that the tensorprocessor is to perform on respective ones of a plurality of tensors;and a second circuitry configured to store each respective tensor in atensor register file dedicated to a tensor operation calculator that isconfigured to perform the type of tensor operation the tensor processoris to perform on the respective tensor; wherein the second circuitry isfurther configured to store multiple separate copies of tensors, forwhich the tensor processor is to perform the first type of tensoroperation, in each of the dedicated tensor register files that areassociated with the multiple instances of the tensor operationcalculators configured to perform the first type of tensor operation, tothe exclusion of others of the plurality of tensor register files. 15.The tensor processor of claim 14, further comprising circuitryconfigured to: decode machine instructions to perform respective ones ofthe types of tensor operations; and control read access to the pluralityof tensor register files based on the type of tensor operation that amachine instruction is to perform when executed in one of the tensoroperation calculators.
 16. The tensor processor of claim 14, furthercomprising circuitry configured to: only allow read access to a first ofthe tensor register files that is dedicated to a first of the tensoroperation calculators responsive to executing a first type of tensoroperation in the first tensor operation calculator; and only allow readaccess to a second of the tensor register files that is dedicated to asecond of the tensor operation calculators responsive to executing asecond type of tensor operation in the second tensor operationcalculator.
 17. The tensor processor of claim 14, wherein to determine atype of tensor operation that the tensor processor is to perform onrespective ones of a plurality of tensors the circuitry is configuredto: access an identifier of a type of tensor register file from amachine instruction to load a tensor, wherein the identifier isassociated with a type of tensor operation.
 18. The tensor processor ofclaim 14, further comprising circuitry configured to: decode a machineinstruction to determine a type of tensor operation to be performed inthe tensor processor; determine an instance of a tensor operationcalculator to execute the machine instruction; access a tensor from atensor register file that is dedicated to the instance of the tensoroperation calculator; and execute the type of tensor operation using theaccessed tensor.
 19. The apparatus of claim 1, wherein the logic isfurther configured to: store a matrix in a tensor register filededicated to a matrix-vector operation calculator; and store a vector ina tensor register file dedicated to a vector-vector operationcalculator.
 20. The tensor processor of claim 14, wherein the secondcircuitry is further configured to: store a matrix in a tensor registerfile dedicated to a matrix-vector operation calculator; and store avector in a tensor register file dedicated to a vector-vector operationcalculator.